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ABSTRACT 

Refutation texts, which are texts that rebut scientific misconceptions and explain the normative concept, can be effective 
devices for addressing misconceptions and affecting conceptual change. However, few, if any, refutation texts specifically 
related to climate change have been validated for effectiveness. In this project, we developed and tested three refutation texts 
related to climate change. The three texts, which are freely available for other educators and researchers to use, relate to (1) the 
enhanced greenhouse effect, (2) the distinction between climate and weather, and (3) the nature of ozone depletion and how 
it is not a major contributor to global warming. Participants in the study consisted of 146 undergraduates enrolled in various 
educational psychology courses. They were randomly divided into experimental and control groups. All participants 
completed a knowledge pretest, posttest, and delayed (2 wk) posttest. Experimental-group participants read one of the 
refutation texts, whereas control subjects read an unrelated text. Based on repeated-measures ANOVAs, results indicated that 
the refutation texts were generally effective in increasing knowledge and addressing the misconceptions that they were 
designed to address, although results were mixed for the weather versus climate text. Furthermore, there was little regression 
at the delayed posttest, except for the weather versus climate text (understanding that climate is usually measured over a 30-y 
period) and for global warming being unrelated to skin cancer. The importance of combining these texts with other 
instructional activities and simulations is discussed. © 2017 National Association of Geoscience Teachers. [DOI: 10.5408/15- 
109.1] 
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PROBLEM AND CONTEXT 

Although misconceptions can develop in a number of 
content areas, they are especially prevalent in the sciences 
and can be extremely hard to overcome (Thijs, 1992; 
Shymansky et al., 1997). Learning about climate change is 
no exception. Despite the overwhelming consensus among 
climate scientists regarding the existence, causes, and effects 
of climate change, and the present public focus, it is still a 
controversial topic among much of the public, and miscon¬ 
ceptions abound (McCaffrey and Buhr, 2008; Miller, 2012). 

Research indicates that students may generalize major 
environmental issues and are often unsuccessful at distin¬ 
guishing the problems from each other and separating the 
distinct mechanisms involved in each issue (Francis et al., 
1993; Keller, 2006). For instance, students often confuse the 
natural greenhouse effect, global warming, and ozone 
depletion (Dunlop, 1998; Keller, 2006). Between reliance 
on their own experiences to make sense of scientific 
phenomena and their difficulty differentiating between 
weather and climate, it is not uncommon for students to 
use weather as "proof" for, or against, climate change 
(Vosniadou and Brewer, 1992; Sinatra et al., 2012). Students 
need some understanding of climate change-related subjects 
to overcome some misconceptions they may have acquired 
through the media, personal experience, and in other ways. 

As indicated by Lombardi et al. (2013), the science of 
climate change is complex. People hold inaccurate concep¬ 
tions about climate systems in general; for example, they 
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hold misconceptions regarding the distinction between 
climate and weather, as well as regarding the causes, effects, 
and mitigation of climate change (Choi et al., 2010; Dutt and 
Gonzalez, 2012; Lombardi and Sinatra, 2012). Therefore, it 
may be necessary, in climate science, as in much of science 
education, to use instructional tools specially designed to 
counter misconceptions and encourage students to think 
about other perspectives or alternate conceptions (Vosnia¬ 
dou, 2008; Sinatra and Broughton, 2011). 

One such instructional tool that may be particularly 
effective is refutation text. Traditional expository texts, which 
only explain concepts, are typically the texts used in science 
education, and students often have a hard time understand¬ 
ing them (McKeown et al., 1992). Consequently, they are 
often relatively ineffective at inducing students to change 
their conceptions (McKeown et al., 1992). Refutation texts, 
however, activate common misconceptions, explicitly refute 
them, and then state the correct or preferred conception; 
these texts can, therefore, be very effective for encouraging 
students to change their conceptions (Guzzetti et al., 1993; 
Hynd, 2001). An example of such a text is shown in Table I. 

Three decades of research have overwhelmingly dem¬ 
onstrated that refutation texts are much more effective at 
eliminating misconceptions and changing learners' concep¬ 
tions than traditional expository texts are (Guzzetti et al. 

1993, Hynd and Alvermann, 1986; Diakidoy et al., 2003; 
Broughton and Sinatra, 2010; Tippett, 2010; Ariasi and 
Mason, 2011). Research also indicates that changes in 
conceptual understanding induced through refutation text 
are also more likely to be maintained over time (Hynd et al., 

1994, 1997; Mason and Gava, 2007; Frede, 2008), and that 
refutation texts are generally preferred by learners (Guzzetti 
et al., 1997; Hynd, 2001; Mason et al., 2008). According to 
the coactivation hypothesis (Van den Broek and Kendeou, 
2008), refutation texts work because they activate the 
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TABLE I: Example of refutation text pertaining to ostriches. 

Many people believe that an ostrich will bury its head in the sand when it is in danger. This is not true, however. If ostriches buried 
their heads, they would not be able to breathe! Ostrich chicks may hide from danger by lying with their necks stretched out along 
the ground. Adults may listen for sound with their heads near the ground, or they might run away. 

Note. Text from Tippett, 2004. 


misconception and correct the conception simultaneously; 
therefore, readers are more likely to become aware of the 
discrepancy between them. In addition, the refutation 
statement likely directs the learner's attention toward the 
new information to be learned, leading the learner to use 
cognitive strategies to resolve the conflict and to integrate 
the new, correct information into their understanding of a 
concept (Ariasi and Mason, 2011). 

However, we are unaware of any refutation texts that 
have been published specifically on climate change, except 
for some by Cook et al. (2014), which we discuss in more 
detail below. Although there are Web sites and blogs that 
specifically refute myths about climate change (e.g., Gore, 
2011; Skeptical Science, 2015; Real Climate, 2016), none of 
these sites were specifically designed for classroom use. 

In regard to Cook et al. (2014), undergraduates wrote 
refutation texts as a class assignment in a course examining 
the nature and causes of climate-related misconceptions. 
The three best submissions, all with perfect scores, were 
subsequently published by Cook (2014) in Skeptical Science. 
The misconceptions addressed were (1) increased solar 
activity as an alternative cause of global warming, (2) the 
incidence of hurricanes is not increasing, and (3) many 
scientists in the 1970s predicted a coming ice age. 

Our study builds on Cook's work in adding to the stock 
of publically available refutation texts and empirically 
validating their effectiveness. The misconceptions ad¬ 
dressed by Cook (2014) were specifically tied to misinfor¬ 
mation propagated by climate change skeptics, whereas our 
interest is in common confusions and misunderstandings 
that laypeople have, although these may be indirectly 
exploited by skeptics. McCaffrey and Buhr (2008) outlined 
10 different widespread misconceptions that contribute to 
the public's lack of climate literacy. Our texts specifically 
address three of those misconceptions, namely, that (1) the 
greenhouse effect is entirely caused by humans, (2) climate 
change is primarily caused by a hole in the ozone, and (3) 
weather and climate are the same thing. For empirical 
documentation of the prevalence of these misconceptions, 
see Gautier et al (2006), Gowda et al. (1997), and Rebich 
and Gautier (2005). 

One goal of our work was to evaluate the effectiveness 
of these texts empirically for college undergraduates not 
majoring in science. 

METHOD 

Participants 

There were 146 undergraduate students from a large 
southwestern university who participated in one of three 
experiments to satisfy an introductory educational psychol¬ 
ogy or educational assessment research requirement. Par¬ 
ticipants ranged in age from 17 to 58 y (mean = 24.2, SD = 
6.3). Most of the participants were female (75.9%), and most 
students were juniors (58.3%) with some sophomores 


(20.7%) and seniors (22.1%). Most of the participants were 
white (51.4%). Other ethnicities represented included 
Hispanic/Latino/Chicano (19.9%), Asian/Asian American 
(16.4%), African American (5.5%), biracial (5.5%), and other 
(1.4%). Most students had education majors (70.5%), and 
the average self-reported grade-point average was 3.15 (SD 
= 0.53). Table II shows the breakdown of those character¬ 
istics among the three experiments. 

MATERIALS 

Refutation Texts 

The intervention for the three experiments in this study 
were three separate refutation texts created for climate 
change education for students ranging from ninth grade 
through college level, based on commonly held climate 
change-related misconceptions. (Our studies, however, 
focused specifically on undergraduate students.) The three 
texts related to (1) the enhanced greenhouse effect, e.g., the 
misconceptions that the greenhouse effect is only human 
caused and consists of a layer of pollution or dust that traps 
sunlight; (2) ozone depletion, e.g., misconceptions that the 
ozone hole is an actual hole and that it is causing global 
warming; and (3) weather versus climate, e.g., the miscon¬ 
ceptions that short-term weather events are the same as 
climate change. Table III lists the specific misconceptions 
that each text addressed. 

The three texts are available in the supplementary 
materials to this article (available in the online journal and 
at <http://dx.doi.org/10.5408/15-109sl>). The texts were 
written by the researchers to focus on targeted misconcep¬ 
tions identified by a team consisting of science educators, 
climate change experts, and educational psychologists. 
Input from the climate change experts, science educators, 
and educational psychologists involved in the project was 
incorporated to create pedagogically sound, scientifically 
accurate texts to educate people regarding these topics. The 
weather versus climate text was prepared and reviewed by a 
secondary geoscience teacher. The texts on the greenhouse 
effect and ozone depletion were reviewed for content 
validity by a science specialist with expertise in climate 
change as well as pedagogy and by an educational 
psychologist with expertise in creating and evaluating 
refutation texts. Multiple rounds of revisions were made 
based on the experts' suggestions. The texts were then 
piloted with three to four undergraduates individually, who 
read each sentence and voiced their understanding; this 
process resulted in several minor wording changes to 
eliminate ambiguities. 

The texts specifically "coactivate" both the miscon¬ 
ception and the more-normative conception (Kendeou et 
al., 2011) by describing the misconception ("Some people 
believe"), refuting it ("However, this is not correct") and 
by explaining the more normative conception. Following is 
a brief example from the enhanced greenhouse effect text. 
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TABLE II: Sample size and demographic information by refutation text experiment. 


Variable 

Text 

Overall 

Enhanced Greenhouse Effect 

Weather Versus Climate 

Ozone Depletion 

Participants, No. 

44 

68 

34 

145 

Age 

Mean 

26.3 

23.07 

23.59 

24.15 

SD 

10.7 

4.38 

4.62 

6.34 

Gender 





% Female 

88.6 

76.1 

58.8 

75.9 

% Male 

11.4 

23.9 

41.2 

24.1 

Grade 

% Freshmen 

4.7 

2.9 

2.9 

3.4 

% Sophomores 

20.9 

20.6 

20.6 

20.7 

% Juniors 

51.2 

57.4 

50.0 

53.8 

% Seniors 

23.3 

19.1 

26.5 

22.1 

Ethnicity 

% White 

56.8 

48.5 

50.0 

51.4 

% Hispanic/Latino 

29.5 

14.7 

17.6 

19.9 

% Asian/Asian American 

9.1 

20.6 

17.6 

16.4 

% African American 

2.3 

4.0 

8.8 

5.5 

% Biracial 

2.3 

7.4 

5.9 

5.5 

% Other 

0.0 

2.9 

0.0 

1.4 

% Education majors 

72.7 

73.5 

61.8 

70.5 

Grade-point average 

Mean 

3.10 

3.16 

3.18 

3.15 

SD 

0.65 

0.54 

0.36 

0.53 


People often confuse the greenhouse effect with global 
warming. Many people believe that the greenhouse effect is 
dangerous and created through human activity. You may 
have thought this too. However, this is incorrect. The Earth's 
greenhouse effect is NOT dangerous. It is NOT caused by 
humans. The Earth's greenhouse effect occurs naturally. It 


helps keep the planet's average temperature comfortable for 
humans to live on the Earth. Without a greenhouse effect, 
Earth's average temperature would be about —TF. This is 
about 60°F colder than the normal average temperature. Life 
on Earth might not exist at all without the greenhouse effect. 


TABLE III: Misconceptions addressed by the refutation texts. 


Text 

Misconceptions 

Enhanced greenhouse effect 

1. The greenhouse effect is unnatural and dangerous. 

2. Greenhouse gases are a layer of pollution or dust that trap C0 2 inside (like a greenhouse). 

3. Ozone depletion causes global warming. 

4. There is nothing we can do about global warming. 

Weather versus climate 

5. Climate changes from year to year. 

6. A few extremely cold days in the winter is evidence against climate change. 

7. Wind and sunlight control the climate much more than ocean currents. 

Ozone depletion 

8. Ozone depletion causes global warming. 

9. Global warming causes ozone depletion. 

10. The ozone hole is an actual hole in the Earth's atmosphere. 

11. Increased levels of ultraviolet radiation coming into the Earth's atmosphere from ozone depletion 
heat up the Earth's climate. 

12. Global warming is causing increased levels of skin cancer. 
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The text then goes on to differentiate the natural and 
enhanced greenhouse effects. 

In writing the texts, we had to make some difficult 
decisions as to what to include, given that the texts were 
being written for general education students. For example, 
the text on the ozone hole does not mention that 
chlorofluorocarbons and hydrochlorofluorocarbons are 
greenhouse gases. The decision not to discuss these was 
an intentional one made on pedagogical grounds to avoid 
making the text overly complex and to help separate the 
association of the ozone hole from causes with greater 
impacts on the enhanced greenhouse effect. 

In addition, we chose to use the classical definition of 
climate specified by the World Meteorological Organization 
(2016) as an average over at least 30 y, rather than simply 
"over a long period of time." Although we recognize that 
some researchers use a different time frame relative to their 
purposes, e.g., measuring climate over thousands of years in 
studying past ice ages, or only over 20 y for some variables in 
the Intergovernmental Panel on Climate Change (IPCC) 
reports (Baede, 2015), for our purposes, it was important to 
define climate as unambiguously as possible. The classical 
definition is a widely used one (e.g., see U.S. Global Change 
Research Program—Climate Change Science Program, 
2009). Each of the three texts was assessed for readability. 
The texts were found to have a Flesch-Kincaid grade level 
ranging from 8.2 to 9.0 and Flesch reading ease scores 
ranging from 54.0 to 58.9, indicating that each text was 
relatively easy to read. A pilot study was also conducted to 
examine the comprehensibility of each text; revisions to each 
text were then made based on the results of the pilot. The 
texts ranged in length from 1,009 to 1,016 words and 
addressed three to five common misconceptions each. 

Knowledge Measures 

Knowledge measures, consisting of multiple-choice, 
true/false, and open-ended constructed-response questions, 
were created for each text based on the misconceptions 
addressed and the scientific information contained in the 
texts. Each assessment was reviewed for content validity by a 
science educator with expertise in climate change. Overall, 
each assessment contained from three to five constructed- 
response questions, 4-13 multiple-choice questions, and 
zero to six true/false questions. The assessments and scoring 
rubrics are included in the supplementary materials (avail¬ 
able in the online journal and at <http://dx.doi.org/10.5408/ 
15-109s2>). 

In aggregating the data, we generated a score for the 
selected-response questions, based on the number of correct 
answers, and a separate score for the constructed-response 
questions. For the selected-response scores, reliabilities (a) 
were 0.80, 0.68, and 0.88 for the enhanced greenhouse, 
weather versus climate, and ozone-depletion texts, respec¬ 
tively. Based on a review of the questions for possible 
ambiguities, three questions were deleted from the analysis 
of the weather versus climate text: Question 4 of the 
multiple-choice questions and Questions 1 and 3 of the true/ 
false items. 

For the constructed-response questions, scoring rubrics 
were developed for each question and then refined during 
coding to capture student ideas not contained in the initial 
rubrics, with previously scored responses then rescored. In 


this way, a student could receive credit if they mentioned a 
legitimate line of reasoning that was not in the text. 

Each constructed-response question was scored by two 
raters working independently; the raters did not know which 
condition or time (e.g., pretest or posttest) was associated 
with each answer. The raters periodically met to discuss and 
resolve discrepancies. Interrater reliability ranged from 0.76 
to 0.99 (mean = 0.89), except for the mechanism component 
of Question 1 of the greenhouse-effect text, for which the 
reliability was 0.56 or 0.80 within one-quarter point. These 
figures were based on samples of 22%-66% of all responses, 
depending on the question and how quickly the two raters 
could achieve adequate agreement. All the responses were 
then rescored. Because the range on each rubric varied (e.g., 
0-3, 0-4), some scores were multiplied by an adjustment 
factor, so ranges would be equivalent, e.g., scores on a 0-3 
rubric would be multiplied by 1.33, so the range would be 0- 
4. The scores on all constructed-response questions for a 
particular text were then summed to compute a single 
composite score for the constructed-response questions. 
These were not combined with the scores for the selected- 
response questions because the two different types of scores 
were scaled differently and reflected somewhat different 
competencies, specifically, written production and deeper 
conceptual understanding versus knowledge recognition. 

Procedures 

Each student participated in two survey sessions, either 
in person or online for the weather versus climate text. 
During the first session, each student completed a form in 
which they gave their informed consent to take part in the 
research. Participants were then randomly assigned to either 
the experimental group or the control group. Students then 
completed a demographics survey and a knowledge 
assessment. Next, participants in the experimental group 
were asked to read a refutation text presenting the scientific 
conception of the phenomenon, whereas participants in the 
control group were asked to read a traditional expository text 
presenting the scientific conception of an unrelated phe¬ 
nomenon. Participants then completed the knowledge- 
assessment survey again. At the second session, which took 
place 2 wk after the first session, students again completed 
the knowledge-assessment survey. The reason for the 
second session was to conduct a delayed posttest of 
enduring conceptual change. 

RESULTS 

Table IV shows the overall means and standard 
deviations of the knowledge measures for the pretest, 
posttest, and delayed posttest. A mixed model 2x3 
repeated-measures ANOVA was then performed to examine 
the effectiveness of each refutation text compared with the 
control text. The results are shown in Table V. The time-by- 
treatment interactions show whether there was statistically 
more growth in one group than there was in the other. The 
F-values are used to compute statistical significance; a large 
value indicates that there was a large amount of variation 
among the means when compared with the within-group 
variation. 

Plots of the means are shown in Fig. 1 for the selected- 
response questions and Fig. 2 for the constructed-response 
questions. Because students can forget knowledge over time, 
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TABLE IV: Means (and SD) of scores for each test by question type 1 



Greenhouse Effect 

Weather Versus Climate 

Ozone Depletion 

No. 

Mean 

SD 

No. 

Mean 

SD 

No. 

Mean 

SD 

Selected response 

Pretest 

40 

4.68 

(2.32) 

68 

5.74 

(1.71) 

31 

5.65 

(1.74) 

Posttest 

40 

7.03 

(2.97) 

68 

6.19 

(1.99) 

31 

6.94 

(2.71) 

Delayed posttest 2 

40 

7.10 

(2.49) 

54 

6.46 

(1.78) 

31 

6.45 

(3.05) 

Constructed response 

Pretest 

44 

2.68 

(1.47) 

66 

3.33 

(2.26) 

33 

1.29 

(1.80) 

Posttest 

44 

4.13 

(2.06) 

68 

5.44 

(3.47) 

33 

2.62 

(2.28) 

Delayed posttest 2 

40 

3.88 

(1.84) 

53 

5.19 

(3.51) 

32 

2.88 

(2.23) 


1 Scores are for the entire sample. Selected response includes multiple-choice and true/false questions; scores were computed out of 13 possible points for the 
greenhouse text, seven for the weather versus climate, and 10 for ozone depletion. Constructed-response questions were open-ended; values shown are 
averages of the means for three questions for the greenhouse text, five for weather versus climate, and three for ozone depletion. The scores on each 
constructed-response question were placed on a standard three-point scale (four points for the greenhouse text), and then, the totals computed. 

2 Reflects a 2-wk delay. 


the statistical models were quadratic, allowing the delayed 
posttest means to be lower than the first posttest means, as 
shown in Figs. 1 and 2. 

For the enhanced greenhouse effect and ozone deple¬ 
tion texts, there was significantly more knowledge gained by 
the experimental group than there was by the control group. 
There were significant effects for the weather versus climate 
text only for the constructed-response questions; the 
selected-response analysis showed only marginal change 
in the means (p = 0.13). Using the Cohen (1988) guidelines 
for partial r\ 2 (0.06 > medium effect; 0.13 > large effect), all 
the statistically significant effect sizes were large or very 
large. Except as noted above, the differences between the 
experimental and control groups were all statistically 
significant at the posttest and delayed posttest time points, 
as indicated by the nonoverlapping error bands in the Figs. 1 
and 2. 

For the enhanced greenhouse effect and ozone deple¬ 
tion texts, inspection of the posttest and delayed posttest 
means for the experimental group indicated only a slight loss 
in knowledge by the time of the delayed posttest, specifically 
no more than 13.4%. A notable decline was seen only for the 
weather versus climate text, specifically, of 21.3% on the 
constructed-response questions. 

To better understand these effects, we examined which 
questions in the experimental group showed the strongest 
gains. Our intent was to determine what students were 
"really learning" from these texts and whether misconcep¬ 


tions were being addressed. The questions with the greatest 
learning gains are shown in Table VI. 

The data in the Table VI indicate that there were three 
misconceptions successfully addressed by the refutation text: 
(1) that an ozone hole is a primary contributor to global 
warming, (2) that climate is the same as weather, and (3) 
that the greenhouse effect is entirely human caused. There 
were also strong learning gains on some factual questions 
that were not necessarily tied to misconceptions, e.g., that 
the Earth gives off infrared radiation. Although not shown in 
Table VI because the data were summarized by means rather 
than by percentage of correct responses, students also 
learned that large bodies of water can affect climate. 
Specifically the mean on the constructed-response Question 
5 for the weather versus climate text increased from a score 
of 0.60 at pretest to 0.91 at posttest and to 0.82 on the 
delayed posttest. Although not shown in Table VI, on 
Question 3, which related to the warming of the oceans as 
evidence of climate change, there was not, initially, a 
significant effect, but further analysis showed that this was 
because we gave credit for other legitimate pieces of 
evidence for climate change, such as melting glaciers. When 
we scored the responses just for mentioning the warming of 
the oceans, there was a 30.2 percentage point increase from 
the pretest to the posttest, indicating that many students 
were at least processing the information in the text. These 
gains were preserved on the delayed posttest. 


TABLE V: Times-by-treatment interactions. 


Text 

Question Type 1 

F 

df 

>fp 

Enhanced greenhouse effect 

SR 

23.61*” 

1, 43 

.38 

CR 

10.42** 

1, 38 

.22 

Weather versus climate 

SR 

0.13 

1, 51 

.05 

CR 

9.09** 

1, 50 

.15 

Ozone depletion 

SR 

15.37*** 

1, 29 

.35 

CR 

7.86** 

1, 29 

.21 


Values based on a nonlinear contrast. SR = selected response; CR = constructed response. 
*p < 0.05, **p < 0.01, ***p < 0.001. 
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Enhanced Greenhouse Effect 
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Ozone Depletion 
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FIGURE 1: Mean score over time for selected-response questions, by text. Error bars reflect a 95% Cl. The posttest 
was administered the same day as the pretest; the delayed posttest was administered 2 wk later. 


For the weather versus climate text, what was driving 
the decline in the delayed posttest averages for the aggregate 
scores on the constructed-response questions? The means 
for the individual questions are shown in Fig. 3. The decline 
was largely driven by Question 4, "What time period is 
considered to determine climate change?" and to a lesser 
degree, Question 1, "What is climate?" A high score on 
Question 1 required mentioning the 30-year criterion; fewer 
students did so in the delayed posttest. Some students may 
have forgotten the exact number after a delay of 2 wk; only 
about one-half of the students who showed gains on the 
constructed-response question retained that knowledge by 
the time of the delayed posttest. 


DISCUSSION 

Educational research has shown that students, as well as 
lay adults, harbor various misconceptions about natural 
phenomena (Dole and Sinatra, 1998; Chi, 2008). These 
misconceptions may arise from mental shortcuts and 
heuristics that cause people to jump to conclusions, e.g., 
that a few very cold days are evidence against global 
warming (McCaffrey and Buhr, 2008). The news media, 
which is a primary source of many people's information 
about climate, may also contribute to these misconceptions 
because the media often oversimplifies complex information 
(Gowda et al., 1997). Many of these misconceptions are 
difficult to counter with traditional means of instruction 
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Enhanced Greenhouse Effect 



Weather vs. Climate 



Ozone Depletion 



FIGURE 2: Mean score over time for constructed-response questions, by text. Error bars reflect a 95% Cl. The posttest 
was administered the same day as the pretest; the delayed posttest was administered 2 wk later. 


(McCaffrey and Buhr, 2008). Therefore, it is important for 
geoscience educators to be aware of techniques, particularly 
the use of refutation texts, that have been proven effective in 
countering such misconceptions. 

This article presents our first steps in developing and 
evaluating refutation texts for climate change education. We 
found that our texts were effective in remedying the 
misconceptions they were targeting. There was clear 
evidence that students learned that ozone depletion is not 
the cause of global warming and that there is not an actual 
hole in the Earth's atmosphere, and also, that there is a 
difference between the natural and enhanced greenhouse 
effects. There was also some evidence that students learned 


that climate and weather are different concepts, with climate 
measured over a much longer time, and that the warming of 
the oceans contributes to climate change. Knowledge gains 
were generally preserved after a 2-wk delay, with the degree 
of regression in overall scores ranging from zero to only 
13.4%. The regression on individual questions was seldom 
more than 25%, with two exceptions we discuss later. 

Refutation texts can be powerful because they speak 
directly to misconceptions that students may harbor, putting 
the students into a state of cognitive disequilibrium (Piaget, 
1952). That disequilibrium is then resolved by presenting the 
alternative, normative conception. According to the coac¬ 
tivation hypothesis, coactivation attracts readers' attention to 











30 Nussbaum et al. 


J. Geosci. Educ. 65, 23-34 (2017) 


TABLE VI: Assessment questions with greatest learning gains. 


Question 

Concept 

% Correct 

Pretest 

Posttest 

Delayed Posttest 

Enhanced greenhouse effect text 

4 

The greenhouse effect is natural. 

40.9 

100.0 

100.0 

5 

Greenhouse effect makes Earth fit for human habitation. 

27.3 

86.0 

86.0 

9 

Correct identification of greenhouse gases. 

18.2 

59.1 

54.5 

7 

Earth's surface gives off infrared radiation 

36.4 

81.8 

59.1 

8 

Solar energy equilibrium. 

18.2 

59.1 

50.0 

15 

Greenhouse effect unrelated to ozone depletion. 

27.3 

50.0 

59.1 

16 

Ozone depletion unrelated to greenhouse effect. 

22.7 

54.5 

45.5 

Weather versus climate text 

3 

Nature of prevailing winds. 

18.2 

54.6 

48.4 

1 CR 1 

Climate measured over 30 y. 

21.2 

46.8 

32.3 

4 CR 1 

Climate measured over 30 y. 

4.4 

27.9 

16.2 

Ozone depletion text 

7 

No actual hole in atmosphere. 

35.3 

94.1 

93.3 

8 

Global warming not cause skin cancer. 

52.9 

94.1 

73.3 

4 

Ozone depletion not causing global warming. 

52.9 

100.0 

93.3 

5 

Global warming not causing ozone depletion. 

52.9 

94.1 

86.7 


1 Constructed-response questions: "What is climate?" and "What time period is considered to determine climate change?" Data reflect the percentage of 
students with scores of three, reflecting knowledge of 30-y criterion. 


the discrepancy and allows readers to connect the two 
conceptions and, in some sense, "overwrite" the incorrect 
conception with the correct one. Furthermore, the correct 
conception is explained in simple terms, making it more 
understandable and plausible (Lombardi et al., 2013). 

Cook and Lewandowsky (2011) have argued that 
refutation messages should not start by stating "the myth" 
(especially in a headline) but should instead state and 
explain the core facts. The myth should then be stated only 
briefly before explaining how the myth is misleading. The 
rationale is to avoid a backfire effect in which the myth 
becomes more familiar to the reader and, in turn, more 
accepted. That argument was partially supported in a study 
by Skurnik et al. (2005) in which warning about false claims 



FIGURE 3: Weather versus climate text, with mean 
scores over time for individual constructed-response 
questions. (Ql: What is climate? Q2: What is weather? 
Q3: What evidence indicates climate change is taking 
place? Q4: What time period is considered to determine 
climate change? Q5: What impact do large bodies of 
water have in determining climate change?) 


regarding flu vaccines and other medical topics led to greater 
acceptance of the myths, especially in older adults. However, 
that was not a refutation text study; the independent variable 
was the number of times the participants were told a claim 
was false, not the order of presentation of claims labeled true 
or false. Thus, the Cook and Lewandowsky (2011) recom¬ 
mendation is in need of additional empirical verification. 
Furthermore, our results are not consistent with Cook and 
Lewandowsky's (2011) recommendation because our results 
indicate that a refutation text can be effective by first stating 
and explaining the misconception, at least among college 
students (see also Braasch et al., 2013). One salient 
difference is that the Cook and Lewandowsky (2011) 
recommendation was made in the context of designing 
media messages in which the myth was stated in an 
attention-grabbing headline, whereas our texts were written 
in a more expository style. 

Limitation and Suggestions for Future Research 

There were, however, some limitations to our study, 
which we discuss below, followed by a discussion of the 
implications for practice. 

Limits to Effectiveness 

Although our texts were effective in addressing most 
misconceptions, as mentioned above, there were two 
concepts in which there was significant regression in the 
time between the posttest and the delayed posttest. The first 
concept was that global warming is not linked to skin cancer; 
the mean score declined about 50%. 

The second concept was knowledge of the 30-y criterion 
for measuring climate; only about one-half of the students 
who showed gains on the constructed-response question 
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retained that knowledge by the delayed posttest. It is unclear 
whether the exact figure ("30 y") was less salient after the 
delay and, therefore, mentioned less in the constructed- 
response questions, or the conflation of weather and climate 
was such a strong misconception that it interfered with 
retrieving more newly acquired conceptions. The data 
suggest the former because in only three cases in the 
experimental group did the students regress on Question 1 
to a zero score, reflecting complete confusion between 
weather and climate. In 11 cases showing regression, 
responses on the delayed posttest still reflected that climate 
was measured over an "extended period" or over "several 
years." This result may, therefore, not be highly problematic 
because, as noted in the Method section, not all researchers 
accept the classical 30-y criterion anyway, but what is 
important is that climate not be confused with short-term 
weather patterns and events, such as an unusually cold day 
or season. 

The weather versus climate text was only partially 
effective. There were significant effects on the constructed- 
response mean scores but not on those for the selected- 
response questions. Whereas most students did learn the 
main gist of the text, i.e., how weather differs from climate, 
some of the details of the factors influencing both weather 
and climate were learned less well. We rewrote the text to 
make it potentially more effective and to eliminate some 
minor content inaccuracies. The revised text is contained in 
the supplementary materials along with the initial text 
(available in the online journal and at <http://dx.doi.org/10. 
5408/15-109sl>). 

One other limitation, specifically regarding the en¬ 
hanced greenhouse text, is that there was no evidence that 
students acquired a robust understanding of the greenhouse 
effect. That topic was not one in which students displayed 
strong learning gains. The text most likely needs to be 
supplemented with strong visual models and demonstra¬ 
tions to illustrate a mechanism as complex as the green¬ 
house effect (see, e.g., Nussbaum et al., 2015). Future 
research should address the effect of adding images to the 
text. Likewise, the number of correct responses regarding 
the question pertaining to what can be done to mitigate 
climate change showed some increase at posttest, specifi¬ 
cally from 77% correct at the pretest to 95% at posttest, but 
the percentage of correct responses returned almost to 
baseline by the time of the delayed posttest (81%). The text 
only briefly mentioned some of the mitigation options, and 
many students did not remember all of them after the 2-wk 
delay. Supplemental instruction would, therefore, also be 
needed on that topic. 

Methodological Limitations 

We note several other limitations in our study. First, 
even though many students provided the normative answers 
on our assessments, learning is not the same as acceptance, 
and we cannot know for sure whether these students truly 
accepted the science of climate change. This is a topic that 
could be addressed in future research. Second, the partic¬ 
ipants in our study were not actually enrolled in an 
introductory geoscience course. On the other hand, the 
participants in our sample were somewhat similar to many 
nonmajors taking these courses, although our sample likely 
consisted of proportionately more females. Future work 
should address the impact, if any, of such gender differences. 


Third, the texts presented in this article require additional 
refinement and testing on various other populations, such as 
high-school students. In addition, we encourage other 
researchers to develop refutation texts on other climate 
misconceptions, such as the seven other misconceptions 
outlined in McCaffrey and Buhr (2008; Table I). 

Regarding the texts examined in this study, although we 
found our texts to be effective, the comparison was made 
with an unrelated, expository text and not with a parallel text 
without the refutation component. We had the control 
participants read a text so that the time between the pretest 
and posttest would be equivalent, as would the overall time 
and effort expended by each participant. We used an 
unrelated text so that the overall effectiveness of the texts 
could be assessed. Our goal was not to compare refutation 
and expository text formats because there is substantial 
evidence that the former is superior (Tippett, 2010). 
Nonetheless, future research could use a more topically 
related control text to verify the power of the refutational 
components. 

Finally, although we drew on the expertise of several 
content experts in preparing the text, the weather versus 
climate text was not sufficiently reviewed. Reviewers of 
earlier versions of this manuscript suggested some changes 
be made to that text, as did our climate-change specialist. 
Therefore, a revised weather versus climate text is presented 
in the supplementary materials, along with the initial text 
(available in the online journal and at <http://dx.doi.org/10. 
5408/15-109sl>). The revised text should be used in future 
instructional and evaluation efforts so that it can be further 
validated and refined. 

Implications for Practice 

Although additional research is required to determine 
how best to use and integrate these texts into a geoscience 
curriculum, we can tentatively offer some suggestions based 
on prior research (Tippett, 2010). 

Recommendations for Use 

First, research suggests that it is helpful to activate 
students' prior knowledge before they read a refutation text, 
through either asking adjunct questions or performing a 
demonstration. Guzzetti (1990) found that reading the text 
before conducting a demonstration designed to elicit the 
same concepts was less effective than performing the 
demonstration first. On the other hand, activating prior 
knowledge can also reinforce misconceptions when that 
knowledge is inaccurate, so it is helpful to include a warning 
that some of that information may be faulty (Tippett, 2010; 
Ecker et al., 2011). 

Second, "refutation text may be more effective when 
misconceptions are single ideas rather than complex 
concepts, as less change is required to accommodate new 
information (Chi, 2008)" (Tippett, 2010, p. 965). For more 
complex concepts, refutation text should be supplemented 
with other activities, such as videos, hands-on experiments, 
collaborative argumentation over alternative models (Lom¬ 
bardi et al., 2013), and graphic organizers that evaluate 
alternative models (Chinn and Buckland, 2012). 

We would also add that refutation texts can be easily 
incorporated into professional development (PD) sessions. 
According to McCaffrey and Buhr (2008) and Plutzer et al. 
(2016), many teachers, especially at the middle and high- 
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school levels, possess gaps in their content knowledge and 
harbor misconceptions. These teachers also need instruc¬ 
tional tools that address learning gaps and misconceptions 
in their students. Teacher training that uses refutation texts 
similar to those that we developed can serve both purposes: 
addressing potential deficiencies in teachers' content knowl¬ 
edge and, simultaneously, teaching about conceptual change 
pedagogy (which of course could also address other 
conceptual change techniques). 

Integrating into a Geoscience Curriculum 

Part of the attraction of using a refutation text is that 
research suggests that these texts can often be effective if just 
read by students, without any additional activities or the use 
of best practices (Tippett, 2010). For example, Diakidoy et al. 
(2003) found that, even with students who are not used to 
reading about a complicated topic via text, combining a 
refutation text with their standard instruction (teacher 
presentation, demonstration, and questioning) led to better 
performance than did standard instruction alone or standard 
instruction plus a traditional expository text. A simple way to 
use the texts would be to assign them to be read as 
homework along with appropriate chapters of the textbook 
and then, as part of a lecture-discussion, to question 
students in class about the concepts, so as to assess whether 
the texts were read and understood. In addition, some 
instructors assign excerpts from scientific reports, such as the 
"Frequently Asked Questions" sections in the IPCC (2007) 
report. There are Frequently Asked Question (FAQ) 
passages, for instance, on "What factors determine the 
Earth's climate?" "What is the relationship between climate 
change and weather?" and "What is the greenhouse effect?" 
These FAQ passages, however, are generally not written in a 
refutational style. Instructors could ask students to read a 
refutation text before reading these excerpts to enhance the 
latter's effectiveness. 

In addition, Tippett (2010) recommends that teachers 
"locate and use trade books that use refutation texts." Most 
texts use only an expository format, despite the research 
during the past 20-30 y documenting the effectiveness of a 
refutation format. Therefore, it is a boon when teachers can 
locate instructional materials with a refutation format. When 
that is not possible, or when the materials do not target all 
the misconceptions that teachers desire, supplementing the 
materials with the type of texts contained in this article is an 
option. 

Of course, the refutation texts could likely also be used 
in conjunction with argumentation and classroom discussion 
to further enhance conceptual change. This may be 
important given the politicized nature of the discourse on 
climate change Qacques and Dunlap, 2008) and the existence 
of confirmation bias leading to processing of evidence 
supportive of preexisting conceptions (Taber and Lodge, 
2006). In classrooms in which this appears to be a problem, 
collaborative argumentation, in which students can flexibly 
change sides or find middle ground (Rebich and Gautier, 
2005; Nussbaum, 2008), in conjunction with graphic 
organizers that prompt more-balanced evaluation of oppos¬ 
ing sides (Nussbaum et al., 2007), have been shown to 
facilitate conceptual change. It would be hoped that some 
students would refer back to the refutation texts in critiquing 
the arguments and evidence made by other students or other 
parties, such as politicians. 


Refutation texts could also be used with the type of 
"agnotology-based" curricula advocated by Cook et al. 
(2014). According to Bedford (2010), much of the disconnect 
between the views of ordinary citizens and scientists is due 
to an organized campaign of disinformation that creates 
doubt and confusion in the public's mind 0acques and 
Dunlap, 2008), a process that Proctor (2008) calls agnogenesis. 
The term agnotology refers to the scientific study of agno¬ 
genesis. Bedford (2010) describes how agnotology can be 
incorporated into the geoscience classroom, e.g., by having 
students read and critique works such as the novel, State of 
Fear (Crichton, 2004). Refutation texts could be incorporated 
into an agnotology curriculum early on, when students are 
first learning about the content, and then certain miscon¬ 
ceptions discussed later on when agnogenesis is examined. 
For example, students could discuss how media messages, 
deliberately or otherwise, confuse the distinction between 
weather and climate. Such "active learning" would help to 
elaborate and deepen students' understanding (Nussbaum, 
2008). 

In closing, it is important for geoscience educators to be 
aware of the power of refutation text in addressing scientific 
misconceptions in general, but especially those related to 
climate change, given that the topic is replete with 
misconceptions and agnogenesis. Once the texts are 
developed, they can be quickly read by students and are 
thus a time-efficient instructional tool. Educating students 
about climate change is extremely important, and geoscience 
educators need to use a variety of tools, including refutation 
texts, to help overcome various misconceptions about the 
topic. 
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